Sign Language Recognition Using Kinect

An open source framework for general gesture recognition is presented and tested with isolated signs of sign language. Other than common systems for sign language recognition, this framework makes use of Kinect, a depth camera which makes real-time 3D-reconstruction easily applicable. Recognition is done using hidden Markov models with a continuous observation density. The framework also o ers an easy way of initializing and training new gestures or signs by performing them several times in front of the camera. First results with a recognition rate of 97% show that depth cameras are well-suited for sign language recognition. 1 Motivation and Introduction Using gestures as a natural communication interface between human beings and machines becomes more and more important. This involves controlling computers, as well as processing and translating sign language. When Microsoft released Kinect in November 2010, it was mainly targeted at owners of a Microsoft Xbox 360 console, being advertised as a controller-free gaming experience. The device itself features an RGB camera, a depth sensor and a multiarray microphone, and is capable of tracking users' body movement [9,10]. The interest in the device has been high among developers, and thus, shortly after its release an uno cial open source driver was introduced, followed by many Kinect-based projects and technical demos. Even though Microsoft stated that Kinect that is shipping [2010's] holiday will not support sign language , several demos show how it technically is capable of recognizing signs [11,12,13]. In sign languages, manual features are generally used along with facial expressions and di erent body postures to express words and grammatical features. The manual components can be split into four parameters: handshape, palm orientation, location, and movement. There are similar signs that di er in one of these components only, and thus without considering context, signs can only be recognized precisely when all of these components are known. Nevertheless, a great number of signs can be distinguished by only considering hand location and movement [6]. After the related work part in section 2, we present Dragon y, an open source C++ framework for general gesture recognition that can be used to recognize signs of sign language, utilizing the two above-mentioned manual components. This is achieved by using hidden Markov models that allow training and recognition of isolated signs. In section 4, the framework is tested in several experiments, and an evaluation shows how well it performs when using optimal parameters. A conclusion in section 5 summarizes the achievements of this work and what future work may follow in order to improve it for better sign language recognition.